Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BUG in Series.interpolate: limit_area/limit_direction kwargs with method="pad"/"bfill" have no effect #38106

Merged
merged 45 commits into from
Dec 1, 2020

Conversation

arw2019
Copy link
Member

@arw2019 arw2019 commented Nov 27, 2020

Picking up #31048

cchwala and others added 30 commits January 15, 2020 17:02
This test currently only test `limit_area`.

For `limit_direction` the implementation should later raise an error,
because `pad` and `bfill` both already define a direction. But let's
now first do the implementation of the `limit_area` for `pad`
and `bfill`.
Since methods `pad` and `bfill` in `blocks.interpolate` end up
using `missing.interpolate_2d` which can not (easily) be extended
to support `limit_area`, I introduce the new function
`missing.interpolate_1d_fill`. It is a modified copy of `interpolate_2d`
but only works for 1d data and uses newly introduced function
`_derive_indices_of_nans_to_preserve`, which is now also used in
`missing.interpolate_1d`. It works the same way as the
1D-interpolation functions which are based on scipy-interpolation which
are applied via np.apply_along_axis.
…values` also was changed via appliying `func`
…e used

Test for all forbidden combos of `pad` and `backfill` is included
* black formatting
* typo
…ments

Test on my local machine are not affected by removing the unncessery
arguments `valid` and `invalid`, which are now derived within the
function.
-black formating
-remove variables that are now obsolete
…imit_area_and_limit_direction_with_pad

Manually resolved conflicts:
 * doc/source/whatsnew/v1.1.0.rst: Just a conflicting line
 * pandas/tests/series/test_missing.py: All test for interpolation have
   been moved to a new file pandas/tests/series/methods/test_interpolate.py
   I accepted all changes from upstream, which removed my tests from
   test_missing.py and manually added my test to the new file
   methods/test_interpolate.py
Co-Authored-By: William Ayd <william.ayd@icloud.com>
…imit_area_and_limit_direction_with_pad

I removed most of the code that I have added in this branch and will add it back in step by step in the next commits
Tests should be red now
@jreback jreback added Bug Missing-data np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate labels Nov 29, 2020
@jreback jreback added this to the 1.2 milestone Nov 29, 2020
@arw2019 arw2019 changed the title [WIP] BUG in Series.interpolate: limit_area/limit_direction kwargs with method="pad"/"bfill" have no effect BUG in Series.interpolate: limit_area/limit_direction kwargs with method="pad"/"bfill" have no effect Nov 30, 2020
@arw2019 arw2019 closed this Nov 30, 2020
@arw2019 arw2019 reopened this Nov 30, 2020
@arw2019 arw2019 closed this Nov 30, 2020
@arw2019 arw2019 reopened this Nov 30, 2020
Copy link
Contributor

@jreback jreback left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

great doc-string comment; pin gon green.

pandas/core/missing.py Show resolved Hide resolved

values[invalid] = np.nan
else:
values = interpolate_2d(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is hit? (this is weird because its called from interpolate_2d itself).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

refactored this away (leftover from original PR)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it was hit for one example but values wasn't changed so may as well just drop through

@arw2019 arw2019 closed this Dec 1, 2020
@arw2019 arw2019 reopened this Dec 1, 2020
@jreback
Copy link
Contributor

jreback commented Dec 1, 2020

hmm, precommit is failing.

@jreback
Copy link
Contributor

jreback commented Dec 1, 2020

@simonjayhawkins i expect to merge this for 1.2 but not blocking rc

@jreback jreback merged commit 1829a61 into pandas-dev:master Dec 1, 2020
@jreback
Copy link
Contributor

jreback commented Dec 1, 2020

thanks @arw2019

@arw2019 arw2019 deleted the GH26796-interpolate-kwargs branch December 1, 2020 22:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug Missing-data np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
Projects
None yet
Development

Successfully merging this pull request may close these issues.

limit_area and limit_direction do not have an effect when interpolation method is 'pad'
3 participants